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ABSTRACT 

This study evaluates the quality of cost estimates 
produced by each of four cost progress models--a random walk 
model, the traditional learning curve model, a production rate 
model (fixed-variable model), and a model incorporating both 
learning curve and production rate effects (Bemis production 
rate adjustment model). Emphasis is on assessing the level of 
biaS associated with these models and determining the 
influence of various factors on model performance. Findings 
indicate, on average, the learning curve and Bemis models 
underestimate unit costs, while the random walk and fixed- 
Variable models overestimate unit costs. Different factors are 
evaluated to determine their significance in explaining 
Variations in the bias of unit cost predictions and 
relationships between the significant variables and model cost 
prediction bias are described. Findings indicate the Bemis 
model is superior to the other cost progress models because it 
exhibits the least bias and is not significantly influenced 


(in terms of bias) by variations in the factors considered. 
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I. INTRODUCTION 


A. DEFENSE SPENDING AND PUBLIC INTEREST 

In recent years, spending for national defense has 
increasingly been the focal point of public scrutiny. 59) oo 
trend can be attributed to two underlying causes--the growing 
federal budget deficit, and the problem of cost overruns in 
the acquisition of major weapons systems. 

1. The Federal Budget Deficit 

Between fiscal years 1980 and 1987, the 

Department of Defense annual Budget Authority 

almost doubled, from $143 billion to $281 

billion, with its total exceeding $1 trillion 

Surimg, that “period. This sharp increase 

contributed to rising deficits and aroused 

public concern over the ways defense dollars 

were being spent. [Ref. l:p. 8] 

The Congressional Budget Office's "Economic and Budget 
Outlook: Fiscal Years 1992-1996" indicates that since 1986, 
the defense budget has been on a downward path. Total budget 
authority’ for 1991 was down approximately four percent from 


1990. Moreover, projections for 1992 and 1993 indicate 


further decreases. [{Ref. 3:p. 84] As a result of these 


* Budget authority is "the authority granted to a federal 
agency in an appropriations bill, to enter into commitments that 
result in immediate or future spending." Budget authority is not 
necessarily the amount of money that an agency or department will 
spend during a fiscal year. Instead, it is merely the upper limit 
on the amount of new spending commitments it can make. [Ref. 2:p. 
176] 


trends, it iS more important than ever that the Department of 
Defense (DoD) manages its dollar resources effectively. 

One key area in which costs must be managed 
effectively is the development and procurement of weapons 
systems~. 

During much of the past three decades, constant dollar 
unit costs for major defense systems have grown much 
faster than constant dollar total budgets for these 
systems. The result has been the purchase of smaller 
quantities of new systems, delayed modernization and 
shrinking capabilities. [{Ref. l:p. 10} 
Budget authority and outlays for research, development, test 
and evaluation and procurement accounted for approximately 
forty percent of the national defense budget in 1989 [{Ref. 
4:p. A-146). More effective and efficient utilization of 
these funds could result in significant savings within the 
DoD, thus allowing scarce resources to be applied to other 
important requirements. 
2. Cost Overruns in Weapons Acquisition 

The second underlying cause of increased public 
scrutiny of defense spending is cost overruns. The problem of 
cost overruns in the weapons acquisition process has been a 


major source of consternation and embarrassment for the DoD 


for many years. Numerous researchers and presidential 


- The term weapon system normally refers to the major item of 
equipment and the subsystems, logistical support, software, 
construction and training needed to operate and support it [Ref. 
1l:p. 9}. For purposes of this thesis, weapon system refers only to 
the major item of equipment. 
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commissions during the past thirty years have concluded that 
tens of billions of dollars per year could be saved by 
improving the acquisition process [{Ref. l:p. 32). ne 
Studies (have) repeatedly urged Congress and the Defense 
Department to correct five basic deficiencies: 
1. Setting requirements for the most sophisticated 
systems available, often irrespective of cost; 
2. Changes in program and contract requirements caused by 
changes in military user preferences, leading to 
annual or more frequent changes in program funding 


levels, initiated by Congress and DoD itself; 


3. Lack of incentives for contractors and government 
personnel to reduce program costs; 


4. Failure to develop sufficient numbers of military and 
Civilian personnel with training and experience in 
business management and in dealing with industrial 
firms to oversee the development and production of 
enormous, highly technical industrial programs; and 

5. Underestimated schedules and costs of major programs, 


distorting the decision-making process for the 
allocation of the national budget." [Ref. l:p. 32] 


While progress has been made in each of the areas 
identified above, there remains much room for improvement. 
Major defense procurement programs have repeatedly experienced 
significant unanticipated schedule delays and cost increases. 
More than ninety percent of these programs exceed initial cost 
estimates and, in the majority of cases, the average increase 
in cost has been more than fifty percent, excluding the 


Prreeesperm cuantilty Changes and inflation. ([Ref. l:p. 32-33] 


A cost overrun occurs when the actual cost of a 
program exceeds the estimated cost. Cost overruns typically 
occur when: fair initial cost estimates are made but 
subsequent actual costs are poorly managed and controlled; or 
actual costs are well-managed but initial cost estimates were 
unreallstic.s (Ref. 5S 2c 

There are various reasons why initial cost estimates 
may be unrealistic, particularly unrealistically low. For 
example, institutional incentives may exist both within DoD 
and at government contractors to underestimate costs initially 
in order to get a program approved and started. A second 
possible reason is that processes, techniques or tools for 
creating cost estimates may be weak and provide misleading 
cost forecasts. This thesis addresses a technical question 
related to the latter issue. In particular, this paper 
examines the problem of low/unrealistic cost estimates for 
major weapons programs by analyzing the performance of 


alternative cost estimation models. 


Be COST ESTIMATION MODELS 

Broadly, cost estimation models fall into two categories. 
First, Cost Estimating Relationships (CERs) attempt to explain 
or predict the cost of a "standard" unit of an liven come. 
manufactured or procured in terms of variables reflecting 
qualities, attributes or characteristics of the item. For 


example, aircraft costs may be modeled in terms of speed or 


payload. Second, Cost Progress Models attempt to explain or 
predict changes in unit cost of items over the life of a 
production or procurement program in terms of changes in the 
Circumstances surrounding production or acquisition. For 
example, costs may be expected to depend on the numbers 
acquired and the production rate and thus unit costs may be 
modeled in terms of such variables. The analysis in this 
thesis will focus on the latter type of model, cost progress 
models. 
1. The Learning Curve 
The most commonly used cost progress model is the 

learning curve. 

Learning curves have gained widespread acceptance as a 

tool for planning, analyzing, explaining and predicting 

the behavior of the unit cost of items produced from a 

repetitive production process [Ref. 5:p. 1]. 
Although learning curves were originally developed and applied 
to predict cost and time requirements for the construction of 
ships and aircraft during World War II, they have since been 
applied in many other manufacturing and non-manufacturing 
settings. The learning curve phenomena was first reported by 
T. P. Wright in the Journal of Aeronautical Sciences in 1936. 
Wright observed that, as the quantity of units manufactured 
doubles, the number of direct labor hours/cost associated with 
the production of an individual unit decreases at a uniform 


rate. Moreover, the uniform rate of learning is peculiar to 


the manufacturing process being observed. 
2. Alternative Cost Progress Models 

It is generally acknowledged that other factors, in 
addition to cumulative quantity, influence unit cost and that 
the simple learning curve does not provide a fully adequate 
description of cost behavior. As a result, prior research has 
attempted to improve the simple learning curve model by 
including additional variables. [Ref. 5:p. 2] 

There are now multiple approaches and models available 
for estimating the costs of acquisition programs. Two of the 
most commonly used cost progress model types are the learning 
curve and the production rate adjustment model. Other model 
types include the plateau model, the Stanford-B model, the De 
Jong model and the S-model. These models are differentiated 


by the variables included and the underlying assumptions. 


C. PURPOSE 

It is unclear at present which model type is most 
appropriate for predicting costs under various manufacturing 
conditions. The purpose of this thesis is to evaluate the 
quality of cost estimates produced by each of four cost 
progress models--a random walk model, a learning curve model, 
a production rate model, and a model incorporating both 
learning curve and production rate effects. In *eenduering 
this evaluation, emphasis will be place on assessing the level 


of bias associated with each of these models. 


D. RESEARCH QUESTIONS 

This thesis will address the following research questions: 

1. Primary Research Question 

What is the bias exhibited by available cost progress 
models when predicting the future unit cost of weapons systems 
acquired through a continuing acquisition program? 


2. Subsidiary Research Questions 


a. Are the various available cost progress models 
comparable in terms of bias? 


b. Do particular models result in less biased 
estimates under certain conditions? 


c. What are those conditions that affect the 
performance of the models? 


dad. Can guidelines be established for determining when 

(under what conditions or circumstances) it is 
most appropriate to use a particular model type? 

E. SCOPE, LIMITATIONS AND ASSUMPTIONS 

There are a number of different criteria which can be 
examined in order to assess various aspects of model 
performance. Two of these criteria which are particularly 
important are the accuracy and the level of bias associated 
with a particular model. Accuracy refers to the degree of 
error in a model's prediction, without regard to the direction 
of the error. Bias refers to both the direction and the 
magnitude of error. It indicates whether predictions made 
using a particular model underestimate or overestimate actual 


See Tie wfocus Of this thesis will be Limited to an analysis 


of the bias associated with the various models tested. One 
purpose of the study is to either confirm or disconfirm the 
results of an earlier simulation study by Moses (Learning 
Curve and Rate Adjustment Models: An Investigation of Bias) 
and to determine whether or not those results hold when 
testing real world data. A second purpose is to extend the 


analysis of bias to a larger set of models. 


ae ORGANIZATION OF THESIS 

The remainder of this thesis is organized into four 
chapters. Chapter II provides a review of the literature 
dealing with various cost progress models. Chapter III 
provides a description of the sample, data and measures used 
to conduct the study. Chapter IV provides an analysis of the 
results. Finally, chapter V summarizes the research findings 


and suggests directions for future research. 


II. REVIEW OF PREVIOUS RESEARCH 

Although the progress or learning curve technique of cost 
estimation was discovered prior to World War II, several 
decades passed before statistical studies of this phenomena 
could be readily conducted. This situation can be attributed 
to two underlying problems: (1) the available data were 
frequently too sparse to support statistical analyses, and; 
(2) the sheer volume of calculations and lack of powerful 
computers meant that many laborious hours were required to 
perform operations which can today be performed in a few 
minutes. This second problem had a strong inhibiting effect 
on researchers. As a reSult, it was not until the 1950's that 
Significant statistical studies of the learning curve 
phenomena began to be undertaken. [{Ref. 6:p. 8] 

This chapter reviews the literature dealing with 
development of the learning curve and alternative rate 
adjustment models. In particular, it summarizes the findings 
of some of the major studies which have been conducted in an 
effort to evaluate the relative performance of these models. 
The discussion is organized chronologically into the following 
SEouomngs research prior to 1970, research during the 
seventies, research during the eighties, and research during 


the nineties. 


A. SURVEY OF RESEARCH PRIOR TO 1970 
Ge sleeves 

In 1952, Hirsch [{Ref. 7] published the results of a 
five-year study of a large United States machine builder. The 
purpose of the study was to examine the relationship between 
labor requirements and production volume. Lot size was used 
as a measure of the rate of production based on the existence 
of stable production lot intervals. In addition, practically 
no changes in management or plant and equipment occurred 
during the period of study. Based on the results of his 
study, Hirsch concluded the relation between direct labor 
requirements and lot size was of little consequence in the 
machining and assembling processes. 

2. Cochran 

In 1960, E. B. Cochran [Ref. 8] published an article 
in which he conducted a careful examination of the basic cost 
function--the learning curve--by studying specific 
manufacturing conditions and parameters which relate them to 
cost trends. The purpose of this study was to probe various 
learning curve applications in an effort to develop new 
concepts of learning curve analysis and revised applications. 
Based on the results of his study, Cochran proposed 
refinements to the basic learning curve concept. In 
particular, he suggested that learning may not necessarily 


occur exactly once per unit but instead, may occur either 
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faster or slower. For example, when planning airframe costs 
for a four-engine aircraft, it may be appropriate to consider 
each engine pod as a unit of learning rather than simply 
consider the aircraft as a unit of learning. 

Cochran provided guidelines for accurately identifying 
the unit of learning and identified other factors, in addition 
to worker learning, that would affect the rate of cost 
reduction (e.g., tooling, supervision, parts design and 
shortages). Moreover, he demonstrated how changing rates of 
learning and task changes may result in both shifts of the 
learning curve and non-linear learning. One of the most 
Significant findings presented by Cochran was that: 

Any change in learning rate is equivalent to a change in 
the unit at which standard cost is reached. And this in 
turn generates a major shift in the entire level of cost 
ONCE eer. 319). 

As a result of this finding, Cochran concluded "that 
the determination of learning rate is of major significance in 
Boerecascting and controlling costs." {Ref. 8:p. 319] 

Cochran indicated that the shape of the learning curve 
can be critical in the first 100 units and pointed out the 
fact that there is a wide range of error in straight line 
curves. Accordingly, he suggested that an S-curve pattern may 


be more appropriate than the usual linear learning curve. 


ia 


3. Alchian 

In 1963, Alchian [{Ref. 9} published the results of a 
1949 study conducted for the RAND Corporation. The purpose of 
this study was to examine the similarity of airframe 
manufacturing progress functions among various airframe 
manufacturers. Statistical tests of the similarity of the 
functions among various airframe manufacturers were performed 
using World War II data. In addition, the reliability of 
predictions made with these curves was assessed. 

The results of this study indicated the progress 
functions differed among various airframe types~ and 
manufacturing facilities both in the amount and rate of change 
of required direct labor per pound of airframe. Alchian 
suggested that, for practical purposes, the use of an average 
of individual progress functions may be appropriate. By 
applying this procedure to 22 airframes produced at different 
facilities, the average production error’ was found to be 


3 


Production error was defined as follows: 


Predicted Manhours - Actual Manhours 


PrOGUCCION 222 Gme— 
Actual Manhours 


Direct labor requirements (manhours) for the first 1000 planes were 
predicted for 22 aircraft model--facility combinations using both 
an industry progress curve and an airframe type progress curve. 
The percentage error resulting from the use of each of these curves 
was then computed for each model--facility combination using the 
equation described above. Next, the weighted average error per 
facility (weighted by actual manhours) associated with the use of 
the industry and airframe progress curves was computed for each of 
the four major aircraft model groups examined: bombers, fighters, 
trainers and transports. Based on these figures, the weighted 
average error per facility for all facilities was computed. 


i2 


approximately 25 percent. This same result was obtained for 
the entire output of any particular airframe produced in one 
facility. Specific curves fitted to the past performance of 
a particular manufacturing facility resulted in margins of 
error of approximately 20 percent. 

Alchian examined alternative relationships between 
direct labor per pound of airframe, cumulative number of 
airframes, time and rate of production. "The results cast 
doubts on any of the alternatives being better fits than the 


Meal eprogress curve.'we(/Ref. 9:p. 692] 


B. SURVEY OF RESEARCH DURING THE SEVENTIES 
1. Linder and Wilbourn 

Models which relate costs to various cost-driving 
features or parameters (1l.e., physical/performance parameters 
of the weapons system) typically result in reasonable 
estimates of future recurring unit procurement costs [Ref. 
mOsm. 2771. Nevertheless, Linder and Wilbourn [Ref. 10] 
suggested that in addition to these parameters, various 
characteristics of the procurement program itself (e.g., 
competitive versus sole-source procurement, single-year versus 
multiple-year buys, and low versus high production rates) 
represent cost-driving features which should be accounted for 


in the cost estimating procedure. 


as 


Linder and Wilbourn investigated the effect of 
production rate on recurring missile unit procurement costs. 
In particular, they developed two models to examine how 
production rate influences the position and/or slope of the 
recurring missile hardware cost improvement curve. The first 
model formulated unit recurring cost as a function of a 
constant "annual" production rate. The second model 
formulated unit recurring cost as a function of a variable 
annualized production rate. 

These models were developed based on an analysis of 
the impact of production rate changes on direct and indirect 
costs. Linder and Wilbourn reasoned that higher production 
rates would result in lower fixed costs per unit. Moreover, 
high production rates were expected to lead to a smaller 
percentage increase in indirect costs than direct costs. As 
a result, lower overhead rates should be applied to direct 
costs. Based on the combined effects of lower fixed costs per 
unit and lower applied overhead rates, Linder and Wilbourn 
concluded that unit costs would be reduced at higher 
production rates, ceteris paribus. Moreover, the cost 
improvement curve associated with a high production rate was 
expected to lie below a cost improvement curve associated with 
a lower production rate. 

In assessing the expected effect of production rate on 
the cost improvement curve slope, the researchers made the 


following observations: 
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fe 


At high production rates, average fixed direct costs are 
reduced and variable direct costs constitute a larger 
Dome LOn OL total. direct. costs. 


ae oir ectmGostes fal), indirect costs are reduced at a 
Slower rate such that overhead rates increase. 


These two phenomena have opposite affects on the slope of 
the cost improvement curve. The first one will tend to 
increase the slope of the curve due to the influences of 
learning and other related effects on variable directs costs 
at a high production rate. The latter phenomena will tend to 
flatten the curve as production increases. As a result, "the 
net effect on the cost improvement curve slope depends on the 
relative amounts of direct and indirect costs per unit as well 
as the proportion of each cost category which can be 
considered as fixed or variable." [Ref. 10:p. 280] 

Based on the results of their analyses, Linder and 


Wilbourn reached the following conclusions: 


1. Ceteris paribus, higher production rates result in lower 
unit recurring costs at each production quantity. 


2. Doubling the production rate lowers average unit costs by 
approximately three to seven percent for the quantities 
examined. 


3. Changing the production rate has only a slight influence 
on the slope of the unit cost improvement curve. 


4. The effects discussed above are relatively insensitive to 
changes in the models' parameter values. [Ref. 10:p. 300] 


LS 


2. RAND Studies 
During the 1970's, the RAND® Cerporatvonmecoenaiceecam 
number of studies which examined the relationship between 
production rate and airframe costs. The specific objectives 
and results of two of these studies are summarized below. 
a. Large, Hoffmayer and Kontrovich 

In 1974, the results of a study by Large, Hoffmayer 
and Kontrovich {Ref. 11] were published. "The purpose of this 
study was to investigate the nature, magnitude and causes of 
the influence of production rate on unit Cost [Ret iia 
1ii} Based on the assumption that production rate and unit 
cost vary inversely, the researchers sought to develop an 
estimating model to express the relationship for various 
elements of cost. 

The results of the analysis suggested the effect of 
production rate on manufacturing labor, manufacturing 
materials, tools and engineering could not be predicted with 
confidence. In any specific case, the effect depended ona 
number of factors including how rate changes were achieved, 
the availability of suppliers, the local labor supply, 
management policy, the timing of rate changes, plant capacity, 
and plant backlog. The only element of cost which was found 
to clearly be a function of production rate was overhead. 

Based on their findings, the researchers concluded 


that the influence of production rate on aircraft cost could 
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not be predicted with any degree of confidence. Each case 
should be examined separately and in detail to assess the 
effect of rate. In addition, they suggested that in advanced 
planning studies, rate effects in aircraft production programs 
can be ignored because they are far outweighed by other 
uncertainties. As a result, they indicated that a model that 
does not explicitly consider rate may be preferable for 
advance planning purposes. 
Db. Large, Campbell and Cates 

This study, published in 1976, attempted to derive 
improved parametric equations for estimating the acquisition 
cost of aircraft airframes. Earlier RAND studies had 
indicated variations in cost among different airframes were 
best explained by the quantity produced and aircraft 
characteristics, i.e., airframe unit weight and maximum speed. 
Large, Campbell and Cates [Ref. 12] were unable to identify 
additional characteristics that would make an estimating model 
more flexible and, hence, better able to deal with 
characteristics peculiar to individual aircraft. None of the 
independent variables considered significantly improved the 
reliability of estimates obtained using only weight, speed and 
cumulative quantity. As a result, they suggested that future 
research Colada examines the influence of program 


characteristics on program cost may be more productive. 


17 


3. Smith 

The purpose of this study was to develop and test a 
procedure to assess the effect of production rate changes on 
the direct labor requirements for production of additional 
airframes. Smith [{Ref. 13] proposed a cost model to express 
direct labor hours required as a function of cumulative 
production and production rate. Two approaches to expressing 
the production rate variable were examined: a lot average 
delivery rate, and a lot average manufacturing rate. Data 
from three airframe production programs--the F-4 program, the 
F-102 program, and the KC-135 program--were used te constEaue- 
data sets. These data sets were then examined in the 
cumulative production and production rate cost model using 
regression analysis. 

Production rate was found to be an important factor in 
the cost of airframe production, although its effect was 
subordinate to that of cumulative production. The study 
demonstrated empirically that production rate can be an 
important predictor of variation im unit direct @etager 
requirements. In addition, the results suggested an increase 
in rate up to plant capacity can lead to a decrease in unit 


labor requirements. 
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C. SURVEY OF RESEARCH DURING THE EIGHTIES 
L7aeCrouch 

The purpose of this study was to investigate possible 
sources of bias in the slopes of progress functions which are 
conventionally estimated using the unit cost progress 
function. Crouch [{Ref. 14] asserted that given the fact that 
Mittweosts mare defunction Not Only of cumulative output but 
also of the rate of output per time period, the conventional 
unit-cost progress function has omitted variables. Under 
these circumstances, the use of ordinary least squares 
regression may introduce specification bias into’ the 
estimates. Crouch confirmed this situation mathematically. 

When a variable from the true relation is omitted, a part 
of its influence in explaining the movements of the 
dependent variable is captured by the independent 
Variables which are included. When the omitted variable 
is not correlated with any of the independent variables, 
the coefficients of the included variables are not biased. 
(Ref. 14:p. 42) 

Crouch conducted a pilot study to investigate the 
existence of bias when progress functions are estimated in the 
conventional manner. Unit-cost data (on an annual average 
basis) in constant dollars for ten components of the Hawk 
missile were used. The results of this study indicated that 
when progress functions are estimated in the conventional 


manner, biased estimates of the slopes may be obtained in a 


Significant number of cases. In particular, the results 
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indicated that when returns to scale are constant, the 
estimated learning curve exponent obtained from the 
conventional progress function will be unbiased. However, 
when the returns to scale are not constant, the estimated 
learning curve exponent obtained from the conventional 
progress function will be biased. Negative bias will occur 
when returns to scale are increasing; positive bias will occur 
when returns to scale are decreasing. 
2. Smith 

The objective of this study was to examine the impact 
of production rate changes on the unit cost of weapons 
systems. Smith {Ref. 15] provided a summary of the 
Significant research on the relationship between production 
rate and weapon system cost. In doing so, he analyzed the 
various findings and conclusions and assessed their 
applicability. The results of his research review indicated 
that only rather weak conclusions could be drawn from the 
existing state of knowledge. The principal findings were: 
production rate affects unit costs but, in most cases, not as 
strongly as the learning (cumulative quantity) effect, and; 
the rate effect varies with the weapon system. Smith 
identified four principal cost-rate models--Womer [Ref. 16], 
Washburn (Ref. 17], Linder and Wilbourn [{Ref. 10], and Fazio 
and Russell [{Ref. 18]. Each of these models differed in the 


concept of rate, the number of parameters to estimate, and the 
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range of applicable programs. Nevertheless, none of them was 
considered suitable for use by top level budget planners. 
Peto pee 11 | 

Case studies were conducted using production data for 
Six missile systems. These systems possessed a wide range of 
production characteristics ranging from low volume, labor 
intensive to high volume, highly automated production. The 
results of the empirical research supported the belief that 
under program stretch-out, the most important contributor to 
increased unit costs is an increased overhead allocation. [In 
addition, the idea that labor inefficiency is often a 
relatively unimportant factor in rate adjustments was 
supported. (Ref. 15:pp. 43-44] Based on the results of the 
case studies, Smith concluded that: 

1. A Simple rate-sensitive model which focuses on the effect 
of rate changes on overhead 1S appropriate for the 
programming and budgeting phases. 

2. Long-range planners should disregard rate behavior and 
focus only on military requirements. im Long-term 
planning, production rates and their effects are both 
unpredictable and much less important than other more 
fundamental considerations. (Ref. 15:p. 46] 

3. Balut 
Standard use of learning curve theory involves an 
implicit assumption that overhead is 100 percent variable with 


direct costs. However, plant overhead is actually comprised 


of three components--variable overhead, fixed overhead, and 
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semi-fixed overhead. Variable overhead costs vary with the 
activity rate. They include production-related indirect costs 
that are tied to the number of direct laborers working in the 
plant and the number of units being produced. Fixed overhead 
costs do not vary with the activity rate and are fixed in the 
short-term (e.g., depreciation, insurance, rent, security). 
Semi-fixed overhead is indirect expenses that are partially 
fixed and partially variable such as utilities. Semi-fixed 
overhead costs are typically gathered and reported as pools; 
consequently, the fixed and variable portions are not 
discernable, [| Ref.) 19%: poos -—oF 

The costing of alternative aircraft procurement 
Quantities and rates within the Office of the Secretary of 
Defense (Program Analysis and Evaluation) 1s a two step 
process which considers the heterogeneous nature of overhead. 
First, new average unit prices are derived for each lot, 
consistent with new lot quantities, using the learning curve. 
Then, in order to correct formethe eruencous Underlyiag 
assumption in step one, prices are adjusted to reflect the 
redistribution of fixed overhead resulting from a change in 
the production rate. This second step is referred to as rate 
adjustment. [Ref. 19:p. 65] 

The objective of Balut's [{Ref. 19] study was to 
evaluate the rate adjustment model used by the Office of the 
Secretary of Defense by comparing its predictions to actual 


contractor performance. An improved version of the model 
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derived using actual contractor data was presented. Baris 
illustrated the use of this improved model for situations when 
the contractor has only one program, and for situations when 
the contractor has other ongoing programs. 
4. Bemis 

Unit costs for weapons systems have traditionally been 
projected using the experience curve. This method expresses 
the projected unit cost as a function of cumulative quantity 
produced, regardless of the production rate. Prior research 
into the production rate-cost relationship for weapons systems 
indicates that unit cost varies significantly as a function of 
production rate. These variations are, to a great extent, due 
to the amortization of fixed overhead. [Ref. 20:pp. 84-85] 

Bemis [Ref. 20] proposed a method for estimating 
rate/cost/quantity relationships using system specific cost 
estimates. Only unit-fly-away costs were considered. The 
input data for the model was historical rate/cost/quantity 
Gata wdiOr “ongoing programs, and contractor or in-house 
estimates for new programs. An equation was derived by 
regressing unit cost on cumulative quantity and production 
rate. In most of the cases analyzed in this study, a high 
multiple correlation coefficient (greater than 0.9) was 
obtained. 

Bemis found that when the production rate was stable, 


the experience curve method and the rate/cost/quantity method 
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generated identical unit cost estimates. However, when the 
oroduction rate was variable, lower unit costs were associated 
with higher production rates, and higher unit costs were 
associated with lower production rates. 

Bemls suggested the rate/cost/quantity model could be 
an invaluable tool for approaching "what if" questions in the 
planning and budgeting process. Moreover, he suggested this 
model offered users a means for readily assessing the cost 
effects of program stretchouts, the costs of maintaining a 
warm production base, and the probable effects of program 


acceleration. 


D. SURVEY OF RESEARCH DURING THE NINETIES 
1. Boger and Liao 

In an effort to reflect the effect of production rate 
on the cost of weapons systems, researchers have proposed a 
variety of adjustments to weapons systems cost models. The 
most popular approach has been to augment the traditional 
learning curve by adding a rate term. The resulting learning 
curve is referred to as a rate adjustment model. [Ref. 21:p. 
82] Boger and Liao [Ref. 21] examined the effects of 
different rate measures and cost structures on rate adjustment 
models and illustrated how alternative surrogate production 
rate measures might lead to erroneous conclusions. 

The effect of production rate on unit cost stems from 


economies of scale. As production rates are increased, 
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facilities are utilized more fully and greater specialization 
Saw awor OCCUrSs. Materials costs are reduced because the 
increased volume of materials purchased results in quantity 
discounts. Finally, the increased production volume allows 
fixed overhead charges to be spread over a larger quantity of 
output. Together, these underlying effects work to increase 
efficiency and lower production costs. [{Ref. 21:p. 83] 
Increases in production rate are normally expected to 
result in lower unit costs due to economies of scale; however, 
production rate increases may also lead to diseconomies of 
scale. Such is the case when a plant is operating beyond its 
efficient capacity level. Under these circumstances, factors 
such as over-time pay, lack of skilled labor or the need to 
invest in additional tooling and/or facilities may lead to 
inefficiencies and increased unit costs. [{Ref. 21l:pp. 23-24] 
Because of the difficulty associated with measuring 
production rate, a number of alternative surrogate measures 
have been adopted. The two primary surrogate production rate 
measures are lot size and annual/monthly production quantity. 
Unfortunately, there are weaknesses associated with the use of 


each of these measures. These weaknesses are as follows: 


ime Otis 1 ze: The time required to produce successive, 
comparably-sized lots frequently changes over the life of 
a program. As a result, it is unclear what is being 


measured by the lot size proxy. 


2. Production Quantity/Time Interval: If there is a large 


amount of work-in-progress and the production period is 
long compared to the observation period, units produced 
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in the following time period will actually reflect work 
performed in the previous time period. This can result 
in substantial bias in estimation. 


3. Average Rate for Each Program: This approach may 


understate the erfect of descriptive rate changes. An 
average rate for each program is usually used in cross- 
sectional analysis because the production rate may change 
in a typical préductiene run. 

4. Cumulative Quantity: Cumulative quantity is highly 
correlated with each of the surrogate production rate 
measures discussed above. As a result, analysts have 
been unable to separate statistically the effect of 
learning and production rate. [Ref. 21l:pp. 86-87] 


In order to avoid some of the difficulties associated 
with using these surrogate production rate measures, Boger and 
Liao recommended a ratio of these measures be used. This 
ratio should be keyed to a base production rate. Adoption of 
this approach offers a number of advantages: 

1. Using a surrogate production rate ratio tends to mitigate 
the multicollinearity problem. 

2. Using the rate to which the manufacturer has tooled as 
the base rate provides an indicator of returns to 
Variable inputs. Ratios greater than one indicate 


decreasing returns while ratios less than one indicate 
increasing returns. [Ref. 21:p. 88] 


In addition to examining the problems associated with 
the two primary surrogate rate measures, Boger and Liao 
examined the problem of changing cost structures. These 
changes occur as a result of changes in the production setup. 
Based on their analysis, the researchers concluded that rate 


adjustment models are appropriate only when applied to data 
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collected from plants which have not undergone changes in cost 
Serpucture. 
2. Moses 

In 1991, Moses [{Ref. 5] published the results of a 
study in which he investigated and compared the forecasting 
bias for the learning curve model and a rate adjustment model. 
Specific objectives of the study were as follows: EO 
determine if either the learning curve model or rate 
adjustment model exhibits consistent/systematic bias; to 
determine under what circumstances the two models are biased, 
and; to identify the nature of the bias (i.e., overestimation 
or underestimation of future costs). 

A simulation approach was used to conduct’ the 
research. First, cost series were generated under varying 
Simulated conditions. Then, model parameters were estimated 


by fitting the learning curve and rate adjustment models to 


the cost series. Future costs were predicted using each of 
the models. These predicted future costs were then compared 
with the actual cost to measure bias. Finally, the 


relationship between the level of bias and the simulated 
conditions was investigated using analysis of variance. 

The simulation was conducted by varying seven factors 
which had been found to affect the magnitude of model 
prediction errors in prior research. These seven factors 


wel eC: 
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3) e 


Bias 


where 


Ee 


A 


adjus 


Data History--the number of data points available to 
estimate the parameters for a model. 


Variable Cost Learning Rate--the learning curve exponent. 


Fixed Cost Burden--the proportion of total cost comprised 
of fixed costs. 


Production Rate Trend--the production trend during the 
model estimation period (i.e., gradual growing trend or 
level trend). 


Production Rate Instability/Variance--period-to-period 


fluctuations in the production rate perhaps caused by 
changes in the demand for output or the supply of inputs, 
and annual budget uncertainties. 


Cost Noise Variance--variability in period-to-period 
cost--designed to reflect unsystematic, unanticipated, 
non-recurring random factors (e.g., changes in the cost, 
type or availability of input resources, temporary 
variations in the level of efficiency, and unplanned 
changes in the production process). 


Future Production Level--the production rate planned for 
the future relative to past levels. 
was measured separately for each model as follows: 


PUC - AUC 
AUC 


BIAS = 


UC = Predicted unit cost either the learning curve or 
the rate adjustment model. . 
UC = Actual unit cost generated by the cost function. 


In conducting his analysis, Moses found that the rate 


tment model provided unbiased cost estimates while the 


learning curve consistently underestimated actual costs. The 
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following conclusions regarding learning curve bias were 


drawn: 


1. Learning curve bias stems from the fact that a portion of 
total cost is fixed. The log linear relationship between 
cost and quantity assumed by the learning curve does not 
hold when fixed costs, which are not subject to learning, 
are present. 


2. Bias increases as the proportion of total cost made up of 
fixed costs increases. This relationship holds up to the 
point where fixed costs account for 50 percent of total 
cost; further increases above that level reduce bias. 


3. The production rate during the period of model estimation 
and the production rate during the period for which costs 
are forecast both affect the degree of bias. Bias is 
minimized when there is a consistent production trend 
during these periods. Bias is magnified when there is a 
Slimit Imeproduction rate trend. 

4. The steeper the learning curve slope, the greater the 
level of bias. (This conclusion is based on the 
assumption that the proportions of total cost that are 
fixed or variable remain relatively stable.) 


5. The greater the number of observations, the higher degree 
of bias. 


6. The further into the future predictions are made, the 
greater the degree of bias. 


E. SUMMARY 

The review of previous research conducted in this chapter, 
while comprehensive, is by no means~ all-inclusive. 
Nevertheless, the studies discussed do provide a sound basis 
for assessing the current level of understanding with respect 


to learning curves and rate adjustment models. The following 
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conclusions can be drawn concerning the research efforts to 


date: 


ice. 


The broad objective of most of the studies dealing with 
extensions or modifications to the learning curve has 
been to investigate the relationship between production 
rate and cost to determine whether or not consideration 
of production rate leads to improved cost estimates. 


Findings concerning the importance of considering the 
effect of production rate on unit costs when predicting 
weapons systems costs have been inconclusive. Some 
studies have found production rate to be an important 
predictor of cost variation while other studies found 
production rate to be of little or no significance. 


Production rate affects unit cost but, in most cases, not 
as strongly as the learning (cumulative quantity) effect. 


The production rate effect varies with the weapon system 
and the specific cost elements. 


The influence of production rate on unit cost depends on 
the relative amounts of direct and indirect costs per 
unit and the proportion of each category that can be 
conSidered as fixed or variable. 


Variations in unit cost in response to production rate 
changes stem from economies of scale. These variations 
are, to a great extent, due to the amortization of fixed 
overhead. 


Lower unit costs are associated with high production 
rates and higher unit costs are associated with lower 
production rates. 


The rate adjustment model, as described by Moses (1991), 
provides unbiased cost estimates. Conversely, the 
learning curve consistently underestimates actual costs. 


Learning curve bias stems from the fact that a portion of 
total cost is fixed. 


Learning curve bias is affected by the proportion of 
total cost made up of fixed costs, the production rate 
during both the period of model estimation and the period 
for which costs are forecast, the slope of the learning 
curve, the number of observations, and the time horizon 
for which predictions are being made. 
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Findings by Moses [Ref. 5] concerning the bias associated 
With the traditional learning curve and the rate adjustment 
model are noteworthy. Nevertheless, their significance is 
somewhat tempered by the use of simulated data. The remainder 
of this thesis will focus on confirming or disconfirming the 
results of the Moses [Ref. 5] study and will extend the 
research to include two additional models--a random walk model 


and a model incorporating production rate effects. 


si 


III. SAMPLE, DATA AND VARIABLES 

The purpose of this chapter is to provide an overview of 
the methodology used to conduct this study. The chapter will 
begin with a description of the population from which the 
sample was drawn, the criteria for inclusion of a weapon 
system/procurement program in the sample and the specific raw 
data collected for each weapon system. This will be followed 
by a description of the procedure used to "expand" the sample. 
Next, the four cost progress models that were included in the 
study will be introduced along with the procedure used for 
predicting costs with these models. Finally, the procedure 
used to measure bias in the study will be explained and the 
"demographic variables" and the "condition variables" that 
were selected as likely candidates for being significant 


explainers of bias will be defined. 


A. SELECTION OF THE SAMPLE 
1. Data Sources 
Data used in conducting the study were obtained from 
two sources: the U.S. Military Aircraft Cost Handbook [{Ref. 
22], and; the U.S. Missile Cost Handbook [Ref. 23]. Data 
contained in these handbooks were based on historical Service 
cost data and reflected the annual total obligational 


authority (TOA) flyaway costs for the included programs. 
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Aircraft data fell into five major categories: attack 
aircraft, fighter aircraft, bombers, attack helicopters, and 
patrol aircraft. Missile data fell into four main categories: 
alr-to-air missiles, air-to-surface missiles, surface-to-air 
missiles, and surface-to-surface missiles. All cost data 
contained in these handbooks was normalized to a constant 
fiscal year base (FY-81). Consequently, the consistency of 
this data was ensured for all aircraft and missile programs. 

Weapons systems contained in the aircraft handbook 
were U.S. Navy, Air Force, Marine Corps and Army aircraft. 
These aircraft were combat-oriented and were in the active 
U.S. inventory during the FY 1960-1980 period. Trainers, 
reconnaissance or electronic warfare variants and those 
aircraft produced for foreign military sales were not included 
in the handbook. 

The missile handbook contained cost information for 
U.S. Air Force, Navy, and Army missile programs during the 
period FY 1961-1983. 

Costs reflected in these handbooks were TOA dollars-- 
the amounts budgeted in a specific fiscal year. TOA dollars 
do not reflect actual expenditures in any given fiscal year. 
Nevertheless, TOA dollars do provide an excellent proxy for 
actual dollar expenditures because it is customary within the 
DoD to ensure that expenditures match total obligational 


authority prior to the lapsing of an appropriation. AS a 


9 


result, differences between TOA and actual expenditures are 
normally very small. 
2. Selection Process and Criteria 

Selection of aircraft and missile programs for 
inclusion in this study was based on three criteria: the 
number of plot points or fiscal years, the availability of 
airframe cost data and the completeness of the data. Only 
programs for which five or more fiscal years worth of data was 
available were included in the study. This criterion was 
required to ensure the minimum amount of data necessary for 
statistically fitting the cost progress models. The latter 
two criteria were established to ensure that meaningful 
analyses could be conducted. As a result, only aircraft 
programs which had complete quantity and airframe cost data 
were included in the study. In addition, only missile 
programs for which complete quantity, and guidance and 
control/airframe costs were available were included in the 
Study« 

3. Programs Selected for Study 

Based on the three criteria described above, forty-six 
weapons procurement programs (fourteen missile programs and 
thirty-two aircraft programs) were selected for inclusion in 
this study. The specific aircraft and missile programs that 
comprised the sample are included in Tables 1 and 2 


respectively. 
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TABLE 1 


ae 


i. 


AIRCRAFT PROCUREMENT PROGRAMS INCLUDED IN THE STUDY 


B= 71 


A-37B 


AH-1G 


Le 
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14. 


itaye 


6% 
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2. 
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20. 


Zoli 


OREN 


AH-15S 


AV-8A 


B-47B/E 


F-3A/B/C 
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23% 


24. 
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DU 


28. 


29% 


Boy 
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BZ. 


Bo Lee 


F-101A/B/C 


p=1025 


f= Boo 


F-105B/D 


Saal Seal 


TA pp 2 


MISSILE PROCUREMENT PROGRAMS INCLUDED IN THE STUDY 


1. PHOENIX (AIM-54A) 


2. SIDEWINDER (AIM-9D/G) 


3. SIDEWINDER (AIM-9H) 


4. SIDEWINDER (AIM-9H/L) 


5. SPARROW (AIM-7E) 


6. SPARROW (AIM-7F) 


7. SHRIKE (AGM-45A) 


8. STANDARD ER (RIM-67B/ 


B-1/C-1) 


9. STANDARD ER (RIM-67A) 


10. STANDARD MR (RIM-66A) 


11. STANDARD MR (RIM-66B) 


12. TALOS (RIM-8E) 


13. TARTAR (RIM-24B) 


14. TERRIER (RIM-2E) 


TABLE 2 
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B. COMPILATION OF PROGRAM DATA 
Once the sample or aircraft and missile procurement 
programs had been selected, the following data were obtained 


for each program in the sample: 


1. Program Name--name of the weapon system. 
2. Manufacturer--name of the prime contractor. 


3. Military Service--identified which Service branch(es) 
procured the weapon system. 


4. Program Type--identified the nature of each weapon 
system, 1.e., aircraft or missile. 


5. Mission--identified each weapon system according to its 
primary mission. Alremaft program types were: 
fighter/attack, fighter, attack, bomber, patrol, and attack 
helicopter. Missile program types were: air-to-air, air-to- 
surface, surface-to-air, and surface-to-surface. 


6. Modification--identified whether a particular weapon 
system was an entirely new design or a modification of an 
existing design. 


7. Combined Program--identified whether a procurement 
program included several versions of a given weapon system 
or only one version. 


8. Fiscal Year--identified the fiscal years during which 
quantities of a particular weapon system were procured. 


9. Quantity--identified the number of units of each weapon 
system procured ina given fiscal year. 


10. Aircraft Airframe Cost/Missile Guidance and Control 
Airframe Cost 
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Cr EXPANSION OF THE SAMPLE 

Following the compilation of data described in the 
previous section, the original data sample of forty-six 
programs waS partitioned into 121 cost’ series. Tha 
"expansion of the sample" was accomplished by dividing each 
program cost series into individual, consecutive year-to-date 
cost series. For example, if a particular procurement program 
had cost data available for seven fiscal years, e.g., FY 1970- 
1976, this single cost series could be expanded into four 


Separate cost series as shown below. 


Cost Series # 1: FY 1970-1973 (used to predict 1974 cost) 
Cost Series # 2: FY 1970-1974 (used to predict 1975 cost) 
Cost Series # 3: FY 1970-1975 (used to predict 1976 cost) 


Cost Series # 4: FY 1970-1976 (used to predict 1977 cost) 


The initial cost series for each weapon system included in the 
Sample was comprised of data from the first four fiscal years 
of that particular program (the minimum number of years needed 
to estimate the cost models). Each subsequent cost series for 
a given program was then created by additionally including the 
data point for the next fiscal year in the existing cost 
series. By partitioning the sample size in the study in this 


manner, it was possible to simulate actual usage of the 
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various cost progress models over time and evaluate their 


performance under varying data availability conditions. 


dD. SELECTION OF COST PROGRESS MODELS 

Previous research has demonstrated that three factors are 
particularly useful in predicting the future costs of weapons 
systems: past cost, cumulative quantity, and production rate. 
Numerous models have been introduced in an effort to improve 
the quality of cost estimates over those obtained using the 
traditional learning curve model. These models are 
differentiated by the explanatory variables included and the 
underlying assumptions with regard to the relative importance 
of past cost, cumulative quantity and production rate in 
predicting future costs. 

In addition, these models may be differentiated by the 
period or length of time over which data is observed and used 
in creating a forecast. Some models assume that future cost 
depends only on the most recent cost, quantity and/or 
production rate levels. However, other models assume that 
cost, quantity or production rate levels from early in a 
program's life are also significant and, as a result, 
specifically consider data covering the full production life. 
Regardless of which approach is used, there are both 
advantages and disadvantages associated with the use of data 
comprised of only recent observations or data comprised of 


Peeimiecehiimanad Carlier observations. 


Si, 


Use of recent observations may increase the relevance of model 
results in light of the current situation but sacrifices any 
information reflected in earlier, historical data. In 
addition, models based only on a few recent observations are 
more susceptible to random noise. Conversely, use of older 
observations may reduce the relevance of results or lead to 
results that are not representative of the current situation. 
However, the use of additional data reduces the impact of 
random variance in recent observations. 

Finally, in addition to the two factors discussed above, 
cost progress models may be differentiated by the form of the 
assumed relationship between the dependant variable--cost--and 
the potential explanatory variables--past cost, cumulative 
quantity, and production rate. Relationships may be linear, 
log linear or some other form. 

Four alternative models were selected for inclusion in 
this study: a random walk model, the traditional learning 
curve model, a model which expresses unit cost as a function 
of past cost and production rate (the fixed-variable model) 
and the common rate adjustment model. 

1. Random Walk Model 

The random walk model assumes that future cost 1s a 
function of past cost; however, only the most recent cost is 
relevant. Any deviation from predicted cost is considered 


ranaon deviation. 
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The model is expressed as follows: 


UJ BS 
| 7 ae ~- 4 
kos es an 


phere 
UC. = Predicted unit cost in time period ¢. 
Pee Geva UNTE COSE in Eime period C - 1. 
t =e oer 1 OO. 


This model was selected for inclusion in the study for two 
reasons. First, it exemplifies the cost estimation method 
used by budget programmers when there is only very limited 
historical data. Under these circumstances, future cost 
projections are often based on actual costs in the previous 
period. Second, the random walk model is the most basic and 
naive cost estimation model and, as such, provides a useful 
benchmark for evaluating the performance of other more 
sophisticated models. 
2. Traditional Learning Curve Model 

The second model selected for inclusion in the study 
was the traditional learning curve. This model assumes that 
future cost is a function of both past cost and cumulative 
quantity produced. Moreover, all historical cost data is 


Eonsidered to be relevant. 
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The traditional learning curve model assumes a log linear 
relationship between cost and cumulative quantity and is 


expressed as follows: 


where 


UC. = Incremental unic cost of 2 cemmip cuane:- on 


QO. = Cumulative quantity produced as or period t. 
a = Theoretical first unit cost. 
i = JOG = Learning curve exponent. 
Ikeve, 2 
r = Learning rate. 


The traditional learning curve model was selected for 
inclusion in the study because it is the most widely 
researched cost progress model and represents the foundation 
on which other cost progress model variants are based. 
3. Fixed-Variable Model 

The third model selected for inclusion in the study 
was the fixed-variable model. This model specifically 
addresses the relationship between total unit cost and unit 
variable costs and unit fixed cost. Variable cost per unit 
remains constant; however, fixed cost per unit varies 
depending on the production volume because total fixed cost 1s 


allocated by spreading it over the total volume of output. 
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The fixed-variable model is expressed as follows: 


where 
Ue -Wnmticese iniperiod t. 
a = Variable cost per unit. 
b = Standard fixed cost per unit. 
eee eed UGCGlOn ale = 9. = 0,7 .. 
OF = Production quantity in period t. 
eee vezage Production quantity per period. 


This model was included in the study because it explicitly 

considers the impact of production rate on unit cost through 

the allocation of fixed overhead. In contrast to the 

traditional learning curve model (which includes cumulative 

quantity but not production rate as an explanatory variable), 

the fixed-variable model includes production rate but ignores 
cumulative quantity as an explanatory variable. 
4. Bemis Production Rate Adjustment Model 

The final model included in the study is the most 

widely used rate adjustment model. This model, popularized by 

Bemis [{Ref. 16], was developed by augmenting the traditional 


learning curve model with a production rate term. 
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The model is expressed as follows: 


UCL 


UC. = Predicted unit cost at quantity O ena prodquce1 One a 


per period R. | 
= Cumulative quantity produced as of period t. 


0. 

R. = Production fate if pew caus 
@ = Theoretical first unit cost. 
b = Learning curve exponent. 

c = Production rate exponent. 


The Bemis production rate adjustment model was included in the 
study because it considers both cumulative quantity and 
production rate (in addition to past cost). Hence, it is the 


most comprehensive of the four models in the study. 


Ee ANALYSIS OF BIAS 
1. Unit Cost Prediction 


In order to assess the bias exhibited by the random 
walk model, the traditional learning curve model, the fixed- 
Variable model, and the Bemis model, predicted unit costs were 
estimated by applying each of the four models, in turn, to the 
actual cost series. The following paragraphs describe how 


this procedure was accomplished. 


The random walk model assumes that unit coset en 


next period is the same as unit cost in the current period. 
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Consequently, the predicted unit cost for any period (t) was 
Simply the actual cost from the preceding period (t-1). 
Predicted unit costs for the remaining three models 
were derived in the following manner. First, each of the 
models was separately fit to the initial cost series 
(comprised of the first four fiscal years' data) for each 
weapon system to derive the models' parameters. Then, the 
appropriate data values for cumulative quantity and/or 
production rate from period five were input into each of the 
models to obtain estimated unit costs for period five. These 
two steps were then repeated for each remaining cost series 
for the various weapons systems until predicted costs had been 
derived for every fiscal year for which actual cost data was 
available (e.g., models were next estimated on five years 


worth of data, then used to predict the cost for year Six). 
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2. Bias Measurement-The Dependant Variable 
Once predicted unit costs had been computed for each 
cost series with each model, a measure of bias was determined 


for each prediction as follows: 


PUG Aes 


BIAS = 
AUC 


BIAS = Percentage difference between predicted unit cost 
and actual unit cost. 


PUC = Predicted unit cost from the particular model of 
interest. . 

AUC = Actual unit cost obtained from original unit cost 
data. 


Positive BIAS values indicate a model has overestimated actual 
future cost; negative BIAS values indicate a model has 
underestimated actual future cost. BIAS values of zero 
indicate the predicted cost and actual future cost are 
identical and the associated model is unbiased. 

As in the earlier study conducted by Moses [{Ref. 5], 
BIAS represented the dependent variable in the statistical 
analysis. The basic objective of the study was to determine 
what factors or conditions are useful in explaining variance 


Piet A oe 
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The measures of bias and associated labels for each of the 


four models were as follows: 


Bias for the Random Walk Model: BIASRW 
Bias for the Learning Curve Model: BIASLC 
Bias for the Fixed-Variable Model: BIASFV 


Bias for the Bemis Model: BIASBE 


Given 121 cost series (from the 46 procurement programs), 
there were 121 separate measures of bias for each model. 
3. Explanatory Variable Selection 
Model performance in prediction (i.e., the degree and 


direction of bias) depends on the circumstances in which the 


model is used. Two broad categories of factors which might 
influence model performance and, hence, be useful in 
explaining bias, were identified in the study. The first 


category consisted of "demographic variables" and the second 
category consisted of "condition variables". Together, these 
two groups of variables were the independent variables in the 
statistical analysis. Table 3 summarizes the independent 


variables and their corresponding labels. 
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TAB EE. 3 


INDEPENDENT VARIABLES 


Demographic Variables Labels 
Program fee JE gaye 
Modification MOD 
Military Service MILSERV 
Mission MISSION 


Condition  Varvabples 


Burden BURDEN 

Cost Variance CVAR 

Learning Rate LRATE 

Production Rate Variance RATEVAR 

Future Production Level FUTUPROD 

Past Production Trend BEGTREND, ENDTREND 
Plot Points PLOTPNTS 
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a. Demographic Variables 
The demographic variables describe characteristics 
of the weapons procurement program. Four demographic 


variables were considered in the study: 


eee eo cise ly ae 
Ze OGdrELCatton 
3. Military Service 


Jeeetes' Ss Lon 


These particular demographic variables were 
selected for investigation because they represent readily 
apparent characteristics of the various programs which may 
affect model bias. 
b. Condition Variables 
In addition to the four demographic variables, 


seven condition variables were considered in the study: 


1. Burden 

2. Cost Variance 

3. Learning Rate 

4. Production Rate Variance 
5. Future Production Level 
6. Past Production Trend 


7. Plot Points 
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The label "condition variables" is used here because each of 
these variables in some manner indicates something about 
conditions that existed during the weapon system procurement. 
These are analogous independent variables (factors) to those 
examined by Moses (Ref. 5]. The following paragraphs describe 
each of these variables along with the underlying rationale 
for their inclusion in the study. 
(1) Burden 
Burden (BURDEN) indicates the percentage of 


total unit cost made up of fixed cost. Burden is measured as 





follows: 
BURDEN = p 
+ Db 
where 
a= Variable cost per unit (constant). 


b = Standard fixed cost per unit. 


Note: aandb were estimated parameters from the fixed- 
variable model. 


Burden was included in the study because past research (Linder 
and Wilbourn [Ref. 10], Moses [Ref. 5] and Smunt [Ref. 24]) 
has shown that burden directly influences the impact of 
changes in production rate on Wnit cost. As production Vac. 
increases, the cumulative quantity produced during a period 


increases and the variable cost per unit decreases due to the 
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Meeeetee en lodaningd. in additiom, as production rate in a 
period increases, tfixed cost per unit is reduced as total 
fixed cost is spread over a larger production output. The 
impact of production rate increases as the proportion of total 
cost made up of fixed costs increases. As a result, the 
relative bias of the various models may depend on the level of 
burden. 
(2) Cost Variance 

Cost variance (CVAR) indicates the amount of 
unsystematic variation in unit cost that may result from 
unanticipated, non-recurring, random factors. Examples 
include changes in the cost, type or availability of input 
resources, temporary fluctuations in efficiency, and unplanned 
changes in the production process. 

By assessing the amount of unsystematic 
variation associated with variouS weapons procurement 
programs, it may be possible to determine whether there 1s a 
relationship between cost stability and bias. Neye “eholk 


reason, cost variance was examined in the study. 


ab 


Cost variance was measured as follows: 


CVAR = 


inere 
CVAR = Cost variance. 
Ce = Unit cost for 2 Gi Vemep odueutouge—-@i oa: 
Cag = Average UNITE COSE for all preoduge1 On wee. .eas ene ae 
n = Total number orf production periods. 


(3) Learning Rate 

Learning rate (LRATE) measures the decrease in 
per unit cost that occurs as the quantity of units 
manufactured doubles. It is affected by the type of 
production process and the complexity of the product design. 
Smunt [Ref. 24] found that the degree of learning in the 
underlying production process determines the improvement in 
prediction accuracy that results from including a learning 
parameter inamodel. Learning rate was examined in the study 
to determine the nature of the relationship between learning 


rate and bias for the various models. 


52 


Learning rate was measured as follows: 


where 


Learning rate expressed aS 4a percentage. 
i eee melinme te KDONe bids eat imate | | 
by the traditional learning curve model. 


Oty 
Nou 


(4) Production Rate Variance 
Production rate variance (RATEVAR) reflects the 
severity of period-to-period fluctuations in production rate. 
These fluctuations may result from either changes in demand 


for a particular weapon system or changes in the cost or 


ave labwieety of production inputs. Production rate variance 


was measured as follows: 


ice ~ Ore 


Pave 


RATEVAR = - 


where 
RATEVAR = Rate variance. oe 
om BOUGRGlEY ©f URlGS Proagucea 27 Ele Currence period. 
O AVG = Average quantity of units produced for ail 
periods to date. 
n = Total number of periods to date. 


og 





(5) Future Production Level 

Future production level (FUTUPROD) indicates 
“whether the level of production in the next period {the peered 
for which cost is to be forecast using the model) is high or 
low relative to the current period. Future unit costs are 
predicted using cost progress models fit to past production 
data. As a result, the accuracy of the various models may be 
affected if the production level in the period for which costs 
are being estimated varies significantly from the production 
levels that existed during previous’ periods. Future 
production level was examined in the study to determine 
whether production growth and productionscutbacks atseee same 


tendencies of the various models to over/underestimate unit 


costs. 
Future production level was measured as 
follows: 
On 
FUTUPROD = log (—) 
O, 
where 
PUTUPROD — Pucune progueetom eee . ; 
On = Production level for the next period for which 
costs are being forecast. 
oF = Production level for the last (most recent) 


—~ 


period. 
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Mo) Past Production Trend 

Past production trend indicates the pattern or 
oroduction volume associated with each weapon system included 
in the study. Two variables, BEGTREND and ENDTREND, were used 
to reflect how production volume was changing at the beginning 
and end of each cost series to which models were fit. 
Production rate per period may be low initially in order to 
work out bugs and ensure a stable product design prior to full 
scale production. Alternatively, initial production rate per 
period may be high if the current weapon system represents an 
updated version of an already existing system with only minor 
mecdiieltecatilons. 

Production rate per period may also vary at the 
end of the production run depending on whether the program is 
abruptly cancelled, gradually phased out or continued at some 
minimum level in order to maintain a warm production base. 
These two production trend variables were included in the 
study to determine whether the bias associated with the 


various models was related to past production trend. 
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The production trend variables were measured as follows: 


ae 
SECTREND = ave 7 ©: 
Oryg 
and 
ENDTREND _ 217 Lave 
Qave 
where 


BEGTREND = PIOQUCETON= elena at ele eo nmn. omer 
Ee PEOGUC Ee .en er ene 
ENDTREND = Production trend for the most recent Proauctzea 


period. oat 
OQ; = Quantity produced in first produceion permmeae 
Oe = Quantity produced in most recent period. 
Ose = Average quantity per period produced through 


current period. 


Positive values for BEGTREND and ENDTREND indicate production 
trends that are increasing in volume. 
(7s PLlOtceroInEs 

The final independent variable included in the 
study was the number of plot points (PLOTPNTS). Plot points 
indicates the number of data points available to estimate 
model parameters. The accuracy of thé learning curve, 
fixed-variable, and Bemis models should improve as the amount 
of data available during the model estimation period 


increases. Nevertheless, if a model is inherently biased, 
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increasing the number of plot points will not necessarily 
2liminate the bias associated with the model. The number or 
plot points was considered in the study to determine whether 
or not the bias associated with the various models could be 
explained in terms of the availability of past production 


data. 


F. SUMMARY 

This chapter has provided an overview of the sample and 
variables to be used to investigate bias. The discussion 
began with a description of the sample selection process, and 
a summary of the data collected for each program included in 
the sample. This was followed by a description of the 
procedure used to expand the sample from 46 observations 
(programs) to 121 cost series. Next, the four cost progress 
models included in the study--the random walk model, the 
traditional learning curve model, the fixed-variable model, 
and the Bemis production rate adjustment model were described 
along with the rationale for their selection. The cost 
estimation procedure was then explained and a method for 
measuring bias was introduced. Finally, two categories of 
independent variables--demographic variables and condition 
variables--were introduced to be evaluated as potential 


Seurces OL model bias. 
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IV. ANALYSIS AND FINDINGS 

Chapter III introduced eleven variables--four demographic 
variables and seven condition variables--which might influence 
the performance of alternative cost progress models and, 
hence, be useful in explaining the cost prediction bias of 
these models. This chapter describes the statistical 
procedures that were performed to assess the significance of 
each of these variables in explaining bias. It will begin by 
providing an overview of the statistical tests that were used 
to analyze the variables. This will be followed by a 
presentation of some general findings with respect to the 
performance of the four alternative cost progress models. 
Next, model specific findings regarding the significance of 
each of the eleven explanatory variables will be presented. 
Finally, the results of the analysis will be summarized and 


conclusions will be presented. 


A. STATISTICAL PROCEDURES 

Prior to conducting statistical analyses, the distribution 
of values for each of the dependent and independent variables 
was assessed. The presence of extreme values within the data 
set could unduly influence the outcomes of the statistical 
analyses. Accordingly, variable values which lay beyond three 


standard deviations from the mean for the variable were 
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truncated. Truncation involved replacing extreme values with 
values equal to three times the standard deviation for the 
appropriate variable. This approach was applied because it 
reduces the influence of outliers on the results of the 
statistical analyses without discarding and, consequently, 
ignoring the impact of these observations. 

Once all extreme values had been identified and truncated, 
the dependency of BIAS on the demographic variables was 
evaluated for each model using analysis of variance (ANOVA). 
After determining the significance of the demographic 
variables in explaining BIAS, the significance of the seven 
condition variables and MOD was assessed using both simple and 
multiple linear regression analysis. Finally, three different 
sets of Spearman and Pearson correlation analyses were 
conducted. First, correlation coefficients were computed for 
the measures of BIAS from the four different models. Next, 
correlations between the condition variables (including MOD) 
and BIAS were examined for each model. Then values for the 
condition variables (including MOD) were correlated with each 
other in an effort to detect potential multicollinearity. 

In conducting tests of statistical significance, findings 
with alpha values less than 0.01 were considered significant. 
When analyzing pairwise correlations between the explanatory 
variables, correlation coefficients larger than 0.50 were 
regarded as offering strong evidence that multicollinearity 


might be a problem. 


Oy, 


B. ANOVA RESULTS 

Analysis of variance was performed for each of the four 
cost progress models to determine whether or not prediction 
blas could be explained by any of the demographic variables. 


The following demographic variables were examined: 


ie cE, 
oe MCSE RY 
o.. Miso lon 
4. MOD 


5. TYPE x MOD (interaction variable) 


None of the demographic variables tested were significant in 
explaining the bias of cost predictions made with the random 
walk, fixed-variable or Bemis models. However, MISSION, MOD 
and the TYPE x MOD interaction variable were all found to be 
Significant in their ability to explain variations in the bias 
of learning curve cost predictions. ANOVA results for the 
learning curve model are provided in Table 4. Findings 
concerning the significance of MOD in explaining variations in 
the bias of learning curve cost predictions will be discussed 
later, along with the regression results. Differences in 
learning curve bias due to MISSION and TYPE are simply noted. 
No hypothesis was offered to expect differences in learning 


curve cost prediction bias in relation to MISSION and TYPE. 
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Consequently, findings related to these factors were not 
pursued further but are mentioned here for possible 
investigation in future research. Based on the fact that the 
utility of these variables in explaining model bias was 
limited to the learning curve model, the remainder of the 
study was devoted to examining the significance of the 


condition variables (including MOD) in explaining model bias. 


C. OVERALL MODEL PERFORMANCE 
1. Mean Bias 

Summary results for the average level of bias 
associated with cost predictions made using the random walk, 
learning curve, fixed-variable and Bemis models are provided 
in Table 5. The mean bias of predicted unit costs estimated 
with the four models ranged from -0.008313 for the learning 
curve to 0.375045 for the fixed-variable model. The resuikee 
show that on average, the learning curve underestimated unit 


costs by approximately 0.83% while the Bemis, random walk and 


fixed-variable models overestimated nese costs by 
approximately 2.9%, 4.6% and 37.5% respectively. Bias 


measures for all four models were skewed in the positive 
direction. Hence, measures of the median bias were examined 
in an effort to obtain a more objective assessment of model 


performance. 
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TABLE 5D 


RANISCRDERING OF MODELS BY MAGNITUDE 
OF POSITIVE BIAS (Lowest to Highest) 



















Model Mean Bias Median Standard Skewness 
Bias Deviation 

Learning =O. 008313 =O Vio Ua 02 O. 325325 Brae Ops as % 
Curve 

Bemis On 029472 -Q0.012925 O43 539456 ae all 
Random 0.04:5242 0.015789 0.199953 is S67 2 
Walk 

Fixed- O- 579045 0.188371 0.929000 3.391860 


Variable 
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2. Median Bias 

The median bias of predicted unit costs estimated with 
the four models ranged from -0.060802 for the learning curve 
to 0.188371 for the fixed-variable model. When the median 
blas measurements for the various models were compared, the 
relative ordering of the models according to magnitude of 
positive bias remained the same. However, the magnitude of 
positive bias decreased significantly in all cases. This 
Suggests that some observations had a very large positive 
bias, causing the mean bias to be more positive than the 
median. The direction of bias remained unchanged for all 
models except for the Bemis model. In this case, mean model 
biaS waS approximately 2.9% while median model bias was 
approximately -1.3%. These results seemed to indicate that 
the learning curve and Bemis models underestimate unit cost 
(provide low unit cost estimates) while the random walk model 
and especially the fixed-variable model overestimate unit cost 
(provide high unit cost estimates). Perhaps not 
coincidentally, both the learning curve model and the Bemis 
model use cumulative quantity to predict future cost, while 
neither the random walk model nor the fixed-variable model 


contain a cumulative quantity term. 
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3. Model Bias Correlations 

Both Pearson correlation coefficients and Spearman 
correlation coefficients were computed for the values of bias 
associated with each of the four cost progress models. tine 
results of these correlation analyses are provided in Table 6. 
All Spearman correlation coefficients were positive and 
Significant; correlation values ranged from 0.27220 to 
0.58048. However, these results were not completely confirmed 
by the computed Pearson correlation coefficients. Values 
obtained from the Pearson correlation analysis indicated 
Significant bias correlations for only three of the six paired 
model combinations: BIASRW-BIASLC, BIASRW-BIASBE, and BIASBE- 
BIASFV. Significant Pearson correlation coefficients ranged 
from 0.26401 to .063335. Both correlation analyses indicated 
that the strongest positive correlation existed between random 
walk model bias and learning curve model bias. The second 
highest correlation in both analyses existed between random 
walk model bias and Bemis model bias. The third relationship 
which was correlated and significant in both analyses was 
Bemis model bias and fixed-variable model bias. 

Based on the outcomes from the correlation analyses, 
it was concluded that there is evidence of a positive 


correlation between bias for all paired model combinations. 
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Table 6 


CORRELATION ANALYSES OF MODEL PREDICTION BIAS 


Spearman Correlation Coefficients 


— BIASRW BIASLC BIASFV BIASBE 
BIASRW 1 8O0C.00 0.58048% Owe. © 0.34698* 
Ome Oh eKerelnl 0.0025 0O.00Gm 
BIASLC 0.63335* 1.00009 0; 3 Soke 0.43 75s 
0.0001 G0 LF O/O)(0) 1h G7 0007 
BIASFV -0.02073 -0.04602 1.00000 0.54488% 
Ord 0.0001 
0.35905* 1.00000 
0.0 


Correlation Coefficients/Prob>|R| under H,: Rho=0 

























0.8215 0.6162 








BIASBE 





0.26401% 0.13296 





0.0034 0.1460 0.0001 
Pearson Correlation Coefficients 


t TING Icavresic% <s0010 1 
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That being the case, it seemed reasonable to suspect that, to 
the extent prediction bias was positively correlated among the 
various models, the random walk, learning curve, fixed- 
variable and Bemis models might perform in a similar manner 
under the same circumstances. Moreover, it was possible that 
the significance of various explanatory variables in 
explaining prediction bias might be similar between models 
whose prediction bias was highly correlated. iUin\by eheiclee me ae 
resolve these issues, correlation analyses and regression 
analyses were used to study the relationship between model 
bias and the explanatory variables. The following paragraphs 
provide the results of these analyses for each of the four 


models. 


D. REGRESSION ANALYSES AND RESULTS 

The significance of the seven condition variables and one 
demographic variable--MOD--was assessed using regression 
analysis. First, simple regression analyses were performed to 
independently test the significance of these independent 
variables in explaining prediction bias when considered in 
isolation. Then multiple regression analyses were conducted 
to determine the significance of these same independent 
variables in explaining prediction bias while controlling for 
the effects of the other independent variables. Pairwise 
correlations were additionally determined among both 


independent and dependent variables. The following sections 


OF 


describe the results of these analyses for each of the four 
models. The discussion of the results will be organized by 
model type in the following sections. Summary tables 
containing average bias for particular subsets of the sample 
(Table 7) and correlations (Table 8) are provided hére. They 
will be referred to as the discussion proceeds. Regression 
results will be presented in each section that follows. 
1. Random Walk Model Bias (BIASRW) 

Table 9 provides the results of the multiple 
regression analysis for BIASRW. The results indicate that 
approximately 33% of the variation in random walk model bias 
was explained by the nine independent variables. However, 
only two variables--LRATE and FUTUPROD--were significant in 
explaining variations in the bias of random walk model cost 
predictions. The relationships between random walk model bias 


and each of these variables are depicted in Figure 1. 
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RELATIONSHIP BETWEEN MODEL BIAS AND 
SIGNIFICANT EXPLANATORY VARIABLES 


MODEL / 
INDEPENDENT 
VARIABLE MEAN BIAS FOR EACH LEVEL 


Low‘ MEDIUM? | HIGHS | 
RANDOM WALK: 


| | FUTUPROD 0.199371 
 — a 
CURVE: 
| BURDEN | 


ENDTREND OO 9 3.5 0) a0. 0136380 OFUS 21002 
FUTUPROD =Om7 0490 -0.048520 Og.23 5685 


| FIXED- 
VARIABLE 


mur ROD 0.907445 Omer 7S 1 0.060342 





* Identifies variable values from the first quartile of the 


verltanole "Ss .ad1Sstribpution. 


idenettleame craable “values from the second and third 


quartiles of the variable's distribution. 


* Identifies variable values from the fourth quartile of the 


Varliable's distribution. 
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TABLES 
CORRELATION ANALYSIS OF TDi eee: 
VARIABLES AND MODEL BIAS 


SPEARMAN CORRELATION COEFFICIENTS 


VARIABLES MODEL BIAS 


BIASBE 
PLOTPNTS 0.01313 | 
MOD 0.07265 
BURDEN -0.08670 
[evar | 0.09104 |-0.a1an6* | 0.19307 | 0.19021 
RATEVAR -0.01795 
BEGTREND 0.05199 | 
_ENDTREND 0.00670 
_— >t 


| FUTUPROD | 0.42949% | 0.34583 | -0.28896* | -0.11105 | 


PEARSON CORRELATION COEFFICIENTS 


VARIABLES _ MODEL BIAS 


| BIASBE 

-0.04182 
oD 0.14173 

| BURDEN -0.07563 

RATEVAR 

_BEGTREND 0.02868 | 


| ENDTREND 0.08477 0.07822 Um Us Go9 = OO 16.5el 


iS | : 
[ FUTUPROD j} ©.46612* “' 0.31543% »| -0.57893*% ‘e-0. 18495 


x Indicates a < 0.01 
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LAB ie. 2 


Ube Rate ea Air ols OF RANDOM WALK MODEL BIAS 


Source 


Model 
Error 


Carlo ta.l 


Root MSE 
Dep Mean 
Cy 


Variable 


LHTERCER 
BURDEN 
CVAR 
BRaALe 
RATEVAR 
BEGTREND 
ENDTREND 
FUTUPROD 
BEOURNTS 
MOD 


i] 
T 


— 


Analysis of Variance 


AMO Cares a < 0-01 


ak 


Sum of Mean 
DF Squares Square F Value 
9 1.62871 0.18097 6.709 
96 2.58963 0.02698 
0 4.21833 
0.16424 R-square Ose. 
0.05042 Adj R-sq sco 5 
325.76890 
Parameter Estimates 
_ Parameter Standard 1 fomenlr 

Estimate Error Parameter=0 Prob > 
-0.366140 0 159350056 -2.296 0 
0.159379 Ome 0.57 5 2.450 0 
-0.034241 0.14791824 -0.231 0 
0.497784 0.15651890 2.861 0 
-0.062376 0.12325481 -0.506 0 
0.136147 0.06080772 2.239 0 
0.044053 0.02678170 1.645 0 
0.126599 Oo 0225550) 5.669 0 
-0.016620 0.0093281l2 -1.782 0 
0.048861 0.04634494 1.054 0 


Prob>F 
0.0001 


IT] 


20259 
ae 
.8174 
.0052 
.6140 
ay os. 
053 
.O0001 X 
.0780 
. 2944 


RANDOM WALK BIAS 
0-25 





0.2 | 3 es 








0.05 





LRATE FUTUPROD 


Mi tow | /mepium |!HIGH 


Relationship Between Random Walk Mode! 
Bias and Significant Explanatory Variables 


Figure 1 
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a. LRATE 

PRAREN Was “round@iso bem significant in explaining 
variations in the bias of random walk model cost predictions 
in both the simple and multiple regression analyses. The 
estimated regression coefficients indicated a positive 
relationship between LRATE and BIASRW. Computed Spearman and 
Pearson correlations were inconclusive with regard to the 
nature of the relationship. Figure 1 indicates that the 
random walk model bias was positive for all levels of LRATE. 
However, the relationship between LRATE and BIASRW appears 


counterintuitive. One would expect low (high) values of LRATE 


to be associated with high (low) values of bias. In fact, a 
nonlinear relationship exists. Bias is highest (most 
positive) when the learning rate is at extremes, li.e., when 


the learning rate is steepest or most shallow. Bias is lowest 
when the learning rate is in the middle range, 1.e., where 
most programs likely will fall. 
Bb. FUTUPROD 

In addition to LRATE, FUTUPROD was also found to be 
Significant in explaining random walk bias in both the simple 
and multiple regression analyses. The estimated regression 
coefficient was positive in both analyses thus indicating a 
positive relationship between FUTUPROD and BIASRW. Computed 
Spearman and Pearson correlation coefficients for FUTUPROD and 


BIASRW were both moderately positive and significant, thereby 
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confirming this relationship. The results show that when 
FUTUPROD was low (high) the level of positive bias was low 
(high). This relationship is readily apparent in Figure 1 
which shows clearly that when FUTUPROD was low (high), the 
random walk model underestimated (overestimated) unit costs. 
These results were expected. When FUTUPROD is high (low), unit 
costs are normally lower (higher) in the period being forecast 
because of decreasing (increaSing) variable costs per unit 
(learning effect) and allocation of total fixed costs over a 
larger production volume. 
2. Learning Curve Model Bias (BIASLC) 

Table 10 provides the results of the multiple 
regression analysis for BIASLC. The results indicate that 
approximately 73% of the variation in learning curve 
prediction bias was explained by the nine independent 
variables. Five of the independent variables included in the 
multiple regression analysis were significant in explaining 
variations in the level of bias of learning curve model cost 
predictions. These variables were: BURDEN, LRATE, BEGTREND, 
ENDTREND, and FUTUPROD. In general, the simple regression 
results agreed with the multiple regression results in terms 


of the significance of these variables. 
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MU rarer NAGeSIS OF LEARNING CURVE MODEL BIAS 


Analysis of Variance 


Sum of Mean 
Source DF Squares Square F Value Probst 
Model 9 12.92818 1.43646 S2.090 0.0001 
Error 96 G.29728 0.04476 
C Total 105 17 .22566 
Root Moc Osco? R-square 047-50:5 
Dep Mean = 19) (807 FE Adj R-sa ey 27 
Gav. -272/7.49365 
Parameter Estimates 
; Parameter Standard ipenor HO: 
Variable DF Estimate Error Parameter=0 Prob > ITI 
INTERCEP i ee 2 OS 0.2054662]1 “11.318 Geo O10 
BURDEN 1 Os72917 0.08380563 G.450 0.0001 xX 
CVAR il 12/7763 0.19054602 0.671 0.5041 
LRATE i 516325 0.20162526 11.488 0.0001 xX 
RATEVAR ] -0.204656 0.15877497 -1.289 Neezu0S 
BEGTREND ] 263523 OM076 35157 3.620 O00 05 x 
ENDTREND 1 75378 0.03449978 Bye Wiss 00 Ol x 
FUTUPROD 1 (ei 22591 0.02876904 G.26] OOo Gil x 
SOM ils ] =o l1255 0.01201634 -~0.937 0 ote i SS 
MOD il Ol s7628 0.05970084 Zo 5 OR 0izS3 


“eA TCaces a@ =< U0. OL 
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However, there were two noteworthy differences: (1) ENDT RES 
“was Significant in the multiple regression analysis but was 
not significant in the simple regression analysis; and (2) MOD 
waS Significant in the simple regression analysis but was not 
Significant in the multiple regression analysis. These two 
findings will be discussed in detail later in this section. 
The following paragraphs discuss the results of the regression 
analyses for each of the variables identified as being 
Significant. The relationships between the level of each of 
the significant independent variables and model bias are 
depicted in Figure 2. 
a. BURDEN 

BURDEN was found to be significant in explaining 
variations in model bias in both the multiple and simple 
regression analyses. The estimated multiple regression 
coefficient indicates there was a positive relationship 
between burden and prediction bias for the learning curve. 
This result was confirmed by the significant but relatively 
weak, positive Pearson correlation (See Table 8). These 
findings indicate that when the proportion of total cost made 
up of fixed costs was high (low), the level of positive bias 
was high (low). Figure 2 graphically depicts the relationship 
between learning curve model bias and BURDEN. The graph shows 
that low (high) positive bias was indeed associated with 


low (high) levels of BURDEN. 
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BURDEN LRATE BEGTREND ENDTREND- FUTUPROD 
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Relationship Between Learning Curve Model 
Bias and Significant Explanatory Variables 


Figure 2 
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However, the graph also indicates that negative bias occurred 
at medium BURDEN levels. In other words, when the proportion 
of total cost made up of fixed costs was low, the learning 
curve model slightly overestimated unit costs. In addition, 
when the proportion of total cost made up of fixed costs was 
at a medium or moderate level, the learning curve model 
underestimated unit costs. Finally, when the proportion of 
total cost made up of fixed costs was high, the learning curve 
overestimated unit costs by a moderately large amount. This 
behavior confirms the finding in the earlier study by Moses 
[Ref. 5]. Moses found that: 
Negative bias consistently increases with increases in 
fixed cost burden--up to a point--then negative bias 
decreases with further increases in burden. The turn 
around point for all observations is when burden is 50%. 
[Bete Stp. 27] 
Moses attributes this behavior to the fact that when BURDEN is 
OS all costs are variable and subject to learning. Bie 
addition, when BURDEN is 100%, all costs are fixed and are not 
subject to learning. Under these circumstances the learning 
curve model correctly specifies the "true" underlying cost 
function and no bias will resultsgeAccording to Moses cma. 
results only when costs--some subject to learning and some 


not--are combined. [Ref. 5:p. 28] 
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LRATE was found to be significant in both the 
Simple and multiple regression analyses. Moreover, LRATE was 
the most important variable in terms of ability to explain 
variations in the bias of learning curve model predictions. 
This is evidenced by the fact that its t-value far exceeded 
the t-values for the other explanatory variables. The 
positive multiple regression coefficient indicates there was 
a positive relationship between LRATE and BIASLC. This 
relationship was confirmed by the computed Spearman and 
Pearson correlation coefficients (See Table 8). The Pearson 
correlation coefficient was particularly high and reflected a 
relatively strong, positive, linear correlation between LRATE 
ane ELAS LC. The results indicate that when LRATE was low 
(1.e., a high level of learning was occurring) the level of 
positive bias was low; and, when LRATE was high (i.e., a low 
level of learning was occurring) the level of positive bias 
was high. Figure 2 confirms this relationship. When the 
level of learning was high (i.e., the LRATE was low), the 
learning curve model greatly underestimated unit costs. When 
the level of learning was moderately high (i.e., LRATE was 
medium), the learning curve model underestimated unit costs by 
a relatively small amount. Finally, when the level of 
learning was low (i.e., LRATE was high) the learning curve 


model greatly overestimated unit costs. 
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The findings concerning the relationship between 
LRATE and BIASLC described above differ markedly from the 
results presented in the Moses study [Ref. 5]. Moses found 
that learning rate was not Significant in’ ats “abilic meee 
explain variations in learning curve bias. However, the 
results of the current study indicate that learning rate is 
extremely important in terms of its ability to explain model 
bias. This suggests that the traditional learning curve model 


does not adequately specify the affect of learning on variable 


costs. 
c. BEGTREND 
BEGTREND was found to be significant in both the 
Simple and multiple regression analyses. Nevertheless, the 


computed regression coefficients provided conflicting 
information about the relationship between past production 
trend (measured by BEGTREND) and BIASLC. The simple 
regression results (See Table 11) indicated a negative 
relationship while the multiple regression results indicated 
a positive relationship. 

Computed correlation coefficients for BEGTREND and 
BIASLC were examined to provide another look at the nature of 
the relationship. Both Spearman and Pearson correlation 
coefficients reflected the existence of a moderately weak, 


negative relationship between BEGTREND and BIASLC. 
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UNIVARIATE ANALYSIS OF LEARNING CURVE MODEL BIAS 
(BEGTREND) 


Analysis of Variance 


Sum oT Mean 

Source DF Squares Square F Value Prob>F 
Model l S67 SRS Was ZT 0G 0.0001 
Error 104 13.66673 Bao Gt 
Gato cal 105 17 .22546 

Root MSE Doo 2c5t R-square 0.2066 

Dep Mean -0.00776 Adj R-sq 0.1990 

Cay : -4673.24228 


Parameter Estimates 


' Parameter Standard Tron’ hu: 
Variable DF Estimate Error Parameter=0 Prob > {TI 
INTERCEP 1 0.191679 Geos20423 1 3.683 0.0004 
BEGTREND 1 -0.381017 007521702 -5.204 0.0001 xX 


Pog 1 ee bess e202: 
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This indicates that when the past production trend was 
decreasing (increasing), (i.e., the initial production volume 
per period was above (below) the average production volume per 
period), the level of positive bias was high (low). The 
relationship between BEGTREND and BIASLC is depicted in Figure 
2. The graph shows that when the past production trend was 
decreasing (i.e., BEGTREND was negative) learning curve bias 
was highly positive. In addition, when past production trend 
was relatively stable (i.e., BEGTREND was near zero) learning 
curve bias was negative. Finally, the graph shows that when 
the past production level was increasing (1.e., BEGTREND was 
positive) learning curve bias was highly negative. 

In general, the results indicate that when 
production volume at the beginning of a series of production 
lots (i.e., at the start of a program) starts off low and 
subsequently builds upward to a higher volume, the learning 
curve model has a strong tendency to underestimate future 
costs. In contrast, when initial production volume Startssem: 
high, the learning curve is biased toward overestimating 
future costs. 

The relationship between BEGTREND and BIASLC 
described above is consistent with the results of Moses' study 
in terms of the nature of the relationship. However, Moses 
found that learning curve bias was negative for all production 
trends. This phenomenon was not observed in the current 


study. 
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In an effort to determine the source of the 
cenrlicting results from the regression analyses, correlations 
between BEGTREND and the other independent variables were 
examined. Correlations between the independent variables are 
provided in Table 12. The correlations indicate significant, 
relatively strong relationships existed between BEGTREND and 
CVAR, BEGTREND and DRALE; and BEGTREND and RATEVAR. 
Consequently, there is a strong possibility that 
multicollinearity existed between these variables. SUCTwa 
condition could have contributed to the conflicting regression 
results. 

Another possible factor which could have influenced 
the results is that there may have been interactions between 
BEGTREND and the other variables in the multiple regression 
analysis. Interactions between BURDEN and past production 
trend were identified in the Moses study and were found to be 
Significant in explaining variations in BIASLC. 

d. ENDTREND 

ENDTREND was found to be significant in explaining 
model bias in the multiple regression analysis only. This 
result was thought to be largely the result of interactions 
between ENDTREND and the other independent variables. The 
estimated multiple correlation coefficient indicated a 


positive relationship existed between ENDTREND and BIASLC. 
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However, this relationship could not be confirmed by examining 
the computed correlation coefficients. Neither the Pearson 
correlation coefficient not the Spearman correlation 
coefficient was Significant for the relationship between 
ENDEREND ama = BIASLC. 

The positive relationship between ENDTREND and 
BIASLC implies that when the past production trend (ENDTREND) 
was decreasing (increasing) (i.e., the quantity produced in 


the most recent period was less than (greater than) the 


average quantity produced per period), the level of positive 
bias was low (high). This relationship between ENDTREND and 
BIASLC is shown in Figure 2. The graph shows that when 


ENDTREND was low (decreasing production trend) BIASLC was 
negative. In addition, when ENDTREND was medium (relatively 
stable production trend) BIASLC was less negative. Finally, 
the graph shows that when ENDTREND was high (increasing 
production trend) BIASLC was positive. 

In the most general terms, the results indicate 
that when production volume at the end of a series of 
production lots is declining, the learning curve model has a 
tendency to underestimate future costs. Similarly, when 
production volume at the end of a series of production lots is 
increasing, the learning curve model has a tendency to 
overestimate future costs. Note, however, these tendencies 
are mild, as indicated by the small magnitude of the effects 


in Figure 2. 
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@e@. FUTUPROD 

FUTUPROD was found to be significant in explaining 
model bias in both the simple and multiple regression 
analyses. The estimated regression coefficient was positive 
in both analyses thus indicating a positive relationship 
between FUTUPROD and BIASLC. Computed Spearman and Pearson 
correlation coefficients for FUTUPROD and BIASLC were both 
weakly positive and significant, thereby confirming this 
relationship. The results indicate that when FUTUPROD was 
low (high) the level of bias was low (high). This 
relationship is readily apparent in Figure 2. The graph shows 
that when FUTUPROD was low, BIASLC was highly negative. When 
FUTUPROD was medium (i.e., the production level for the next 
period was comparable with the production level for the last 
period) BIASLC was moderately negative. Finally, when 
FUTUPROD was high, BIASLC was highly positive. 

The relationship between FUTUPROD and BIASLC 
observed in this study coincides almost exactly with the 
relationship described in the Moses study [Ref. 5]. As 
expressed by Moses, this relationship should be expected. 

Higher (lower) future production will result in lower 
(higher) fixed cost, and total cost, per unit, creating a 
tendency toward positive (negative) bias for any cost 
estimate. |(Ref. Gap. 22) 

In general, these findings indicate that the 


learning curve model tends to underestimate unit costs when it 
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PolsCOm tOmoaheadtcCtmcosts £Gr periods in which production 
volume 1S cut back. Conversely, the learning curve model 
tends to overestimate unit costs for periods in which 
production volume is increased. 
£. MOD 

MOD was identified as being significant in 
explaining variations in bias in the simple regression 
analysis (See Table 13); however, MOD was not significant in 
the multiple regression analysis. Both Spearman and Pearson 
correlation coefficients were examined. The correlation 
coefficients indicated a significant, weakly positive 
correlation between MOD and BIASLC. This suggests that the 
learning curve model has a lesser tendency to underestimate 
unit costs when used in predicting costs for modification type 
programs. Examination of the correlations between MOD and the 
other explanatory variables revealed there was a significant, 
moderately strong positive correlation between MOD and LRATE. 
Consequently, it is likely that multicollinearity between MOD 
and LRATE was the source of the lack of significance for MOD 
in the multiple regression analysis. This positive 
correlation between MOD and LRATE seems reasonable. One would 
expect modification programs to exhibit less learning than 


programs involving entirely new designs. 


S7 
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UNIVARIATE 


Source 
Model 
Error 
Cal otal 


ROO ta Mse 
Dep Mean 
Cav 


Variable DF 


INTERCEP 1 
MOD 1 


(MOD) 


Analysis of Variance 


Sum of Mean 


DE Squares Square F Value 
1 ivi 9S52 1.19852 7 ie er a 6 
104 16.02694 Oi 5Gac 
LoS 17 .22546 
02.39:256 R-square 0.0696 
-0.00776 Ady R-sq 0.0606 
=O 67 Uo 
Parameter Estimates 
Parameter Standard IP amele ift's 
Estimate Error Parameter=0 Prob 7 
18) J sya 0.065427 06 -2.385 Ono 
Oo 22q550 U. 0605215 2.789 te a 


* Indicates ars70n08 
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ANALYSIS OF LEARNING CURVE MODEL SEiexs 


Probar 
0.0063 
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3. Fixed-Variable Model Bias (BIASFV) 

Table 14 provides the results of the muitipie 
regression analysis for BIASFV. The results indicated that 
approximately 37% of the variation in fixed-variable model 
blaS waS explained by the nine independent variables. 
However, Omeey one of these Varliables--FUTUPROD--was 
Significant in explaining variations in the bias of fixed- 
variable model cost predictions. FUTUPROD was also found to 
be significant in the simple regression analysis. The 
estimated multiple regression coefficient indicates a negative 
relationship existed between FUTUPROD and BIASFV. This 
relationship is confirmed by the computed Spearman and Pearson 
correlation coefficients for FUTUPROD and BIASFV. Bown 
correlation coefficients indicate the existence of a 
Significant negative correlation. Moreover, the relatively 
large negative Pearson correlation coefficient indicates that 
the correlation between FUTUPROD and BIASFV was fairly linear. 
The relationship between fixed-variable model bias and 
FUTUPROD is depicted in Figure 3. 

The graph confirms that when the level of production 
in the next period was high (low) relative to the most recent 
period, the level of positive bias was low (high). In short, 
the fixed-variable model has a tendency to overestimate future 
costs. This tendency is greatest when the model is used to 
predict costs for periods in which cutbacks in production 


volume occur. 
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TABLE 14 


MULTIVARIATE ANALYSIS. OF FIXED=VARIABLE MIODEEeea 


Analysis of Variance 


Sum oF Mean 
Source DF Squares Square F Value Probor 
Model 9 G2.65513 G.73946 7.647 0.0001 
Error 96 57 .98486 0.6040] 
CG lotall 105 100.63998 
Root MSE Oe 138 R-square 0.49238 
Dep Mean 0.492647 Adj R-sqa 0.3698 
Gan. 182 .25765 
Parameter Estimates 
' Parameter Standard T for HO: 
Variable DF Estimate Error Parameter=0 Prob > {TI 
INTERCEP l 1.627186 0.75474637 ZA 56 00S56 
BURDEN l 0.351109 0.3078462]1 1.141 Onc 5e9 
CVAR l =—06 55/7605 0.69993949 al. 1 oe 0.2365 
LRATE ] =| 572007 0.74063728 aa LS 0.0364 
RATEVAR il -0.016944 0.58323376 -0.029 0.9769 
BEGTREND i -0.224974 0.2877 5617 -0.782 0.49362 
ENDTREND lt O. 010113 0.12672928 0.080 0.9366 
FUTUPROD l =0)./51819 0.105673836 -7.114 0.0001 x 
PLOTPNTS ] 0.049066 0.044614006 Tre 0.2691 
MOD i 0.094664 0.21930124 0.432 0.6670 
* Indicates a < 0.01 
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4. Bemis Rate Adjustment Model Bias (BIASBE) 

Table 15 provides the results of the multiple 
regression analysis for BIASBE. Only approximately 2% of the 
variation in Bemis model prediction bias was explained by the 
nine independent variables. Moreover, none of the variables 
were found to be significant in either the simple regression 
or multiple regression analyses. This means that the Bemis 
model was successful in accounting for the influences of these 
variables. The findings concerning Bemis model performance 
Support the conclusions drawn in the earlier Moses study. 
Moses found that the overall mean bias for all cost 
predictions made with the Bemis rate adjustment model was 
-0.0016. As a result, he concluded that, on average, the rate 
adjustment model exhibits no bias. In addition, Moses 
observed that the absence of bias was evident for all 
treatments across all variables of interest. No significant 


main effects were observed in the ANOVA results. (Ref. 5:p.24] 
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Sum of Mean 
Source DF Squares Square F Value 
Model 9 Prooo1}5 0.14879 1.194 
Error 96 11.96836 0.124967 
C Total 105 95-5.07 51 
Root MSE 0.35309 R-square 0.1006 
Dep Mean 0.03091 Adj R-sqa Or Ore 3 
€.V 1142.35596 
Parameter Estimates 
' Parameter Standard T for HO: 
Variable DF Estimate Error Parameter=0 Prob > 
INTERCEP l 0.097655 0.34289489 0.285 0 
BURDEN l =00 5500 1 0.13986008 -0.393 0 
CVAR l 0.161554 0.31799513 0.508 0 
LRATE l -0.040905 0.33648487 =. 122 0 
RATEVAR i -0 ..424442 0.26497361 -1.602 0 
BEGTREND l 0.090834 0.13072464 0.695 0 
ENDTREND l 0.044540 005757540 0.774 0 
FUTUPROD l —Cnd7 510.44 0.04801158 =] .563 0 
PEOTPNTS l -0.008266 0.02005362 -0.412 0 
MOD ih 0.192540 0.09963251 e935 0 


MULTIVARIATE ANALYSIS OF BEMIS MODEL BIAS 
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ice SUMMARY 

This chapter has described the results or statistical 
analyses of the bias associated with unit cost predictions 
obtained using the random walk, traditional learning curve, 
fixed-variable and Bemis production rate adjustment models. 
The discussion began with a description of the statistical 
procedures used in conducting the study. This was followed by 
a presentation of the findings with respect to overall 
performance of the four cost progress models and an 
examination of the factors thought to be useful in explaining 
Variations in model performance (bias). 

Median bias values for the four models indicated that the 
traditional learning curve model and the Bemis model tend to 
underestimate the unit costs of weapons systems while the 
random walk and fixed-variable models tend to overestimate 
unit costs. In addition, correlation analyses of the cost 
prediction bias associated with each of the models reflected 
a significant positive correlation between predicted unit cost 
bias for all paired model combinations. ANOVA and regression 
analyses were conducted to determine the significance of the 
four demographic variables and eight condition variables 
included in the study. Only three of the demographic 
varlables--MISSION, MOD and TYPE x MOD (interaction 
variable)--were significant in terms of their ability to 


explain variations in the level of model prediction bias. 
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Moreover, the significance of these variables was limited to 
the learning curve model. 

Findings concerning the significance of the eight 
condition variables revealed that the utility of these 


variables in explaining model cost prediction bias varied 


widely among the models. Only two condition variables--LRATE 


and FUTUPROD--were significant in explaining variations in the 
bias of random walk cost predictions. Conversely, five 
condition variables--BURDEN, LRATE, BEGTREND, ENDTREND and 
FUTUPROD--were significant in explaining variations in the 
bias of learning curve model cost predictions. FUTUPROD was 
the only condition variable that was significant in explaining 
variations in the bias of fixed-variable model cost 
predictions. Finally, none of the condition variables was 
significant in explaining variations in the bias of Bemis 
model cost predictions. 

The results of the statistical analyses of learning curve 
cost prediction bias differed somewhat from those obtained by 
Moses [{Ref. 5], particularly with respect to the significance 
of LRATE and PLOTPNTS in explaining variations in cost 
prediction bias. However, results for the Bemis production 
rate adjustment model strongly supported the findings 
presented by Moses. Chapter V will summarize the major 
findings from the current study and will suggest directions 


for future research. 
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Ve. SUMMARY 


A. FINDINGS 
The primary objective of this study was to determine the 
bias of selected cost progress models when predicting the 
future unit cost of weapons systems acquired through a 
continuing acquiSltionmprecran- In addition, the research 
sought to answer the following questions: 
1. Are the various cost progress models comparable in terms 
of bias? 


2. Do particular models result in less biased estimates 
under certain conditions? 


3. What are those conditions that affect the performance of 
the models? 


4. Can guidelines be established for determining when (under 


what conditions or circumstances) it is most appropriate 
to use a particular model type? 


This chapter will address each of these areas by summarizing 
the results of the study. 

Overall findings with respect to the bias of unit cost 
predictions obtained with the random walk, traditional 
learning curve, fixed-variable, and Bemis production rate 
adjustment models were as follows: 


1. On average, the learning curve model underestimated unit 
costs by approximately 6.1%. 


2.6 


2. On average, the Bemis model underestimated unit costs by 
apWwOxIMaeehy 123%. 


3. On average, the random walk model overestimated unit 
GOStSF OY o.62. 


4. On average, the fixed-variable model overestimated unit 
GOsts eye 28s: 


These findings indicate that the four models do indeed differ 
in terms of the direction and magnitude of cost prediction 
bias. 

The influence of various factors on model performance was 
examined by evaluating the significance of four demographic 
variables and eight condition variables in explaining 
variations in the bias of unit cost predictions. Findings 
were that the utility of these variables in explaining model 
cost prediction bias varied widely among the models. ileal 


particular, the following relationships were observed: 


1. Random walk model cost prediction bias is influenced by 
two factors--the learning rate associated with the 
production process, and the production level for the 
future period relative to the most recent period. 


2. When the level of learning is high, the random walk model 
overestimates unit costs by approximately 5.2%. At 
moderately high learning levels, the random walk model 
Overestimates unit costs by only approximately 2.8%. 
However, when the level of learning is low, the random 
walk model overestimates unit cost by approximately 7.5%. 


3. The higher the production level in a future period (the 
period for which unit costs are being forecast) relative 
to the most recent production period, the more the 
random walk model tends to overestimate future cost. 
When the future production level is significantly lower 
than the level in the most recent production period, unit 
costs are underestimated by approximately 6.4%. When the 
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future production level is comparable with the most 


recent period's production level, unit costs are 
overestimated by approximately 2.4%. Finally, when the 


future production level is significantly higher than the 
level in the most recent production period, unit costs 
are overestimated by approximately 20.0%. 


Learning curve model cost prediction bias is influenced 
by the following factors: the percentage of total cost 
made up of fixed costs, the level of learning associated 
with the production process, the past production trend, 
and the future production level. 


When the proportion of total cost made up of fixed costs 
varies, learning curve model bias is affected as follows. 
At low levels, unit costs are overestimated by 
approximately 2.5%; at medium levels, unit costs are 
underestimated by approximately 8.3%; at high levels unit 
costs are overestimated by approximately 11.1%. 


The higher the level of learning, the more positive the 
bias of learning curve model unit cost estimates. When 
the level of learning is high, costs are underestimated 
by approximately 15.2%. When the level of learning is 
moderately etoih.; costs are underestimated by 
approximately 5.4%. When the level of learning is low, 
costs are overestimated by approximately 22.0%. 


When the production volume per period is decreasing at 
the beginning of a series of production lots, the 
learning curve model overestimates unit costs’ by 
approximately 19.5%. When the beginning production trend 


is relatively stable, the learning curve - model 
underestimates unit costs by approximately 5.6%. When 


the production volume per period at the beginning of a 
series of production lots is increasing, the learning 
curve model underestimates unit costs by approximately 
2 a2. 


When the production volume per period is decreasing at 
the end of a series of production lots, the learning 
curve model underestimates unit costs by approximately 
3.9%. When the ending production trend is relatively 
Stable, the learning curve model underestimates unit 
costs by approximately 1.4%. When the production volume 
per period is increasing at the end of a series of 
PECGdUuctI Cone oesr the learning curve model will 
overestimate unit costs by approximately 3.2%. 
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9. There 1S a positive relationship between the level of 
production in a future period (relative to the most 
recent period) and learning curve model cost prediction 
bias. When the level of production in a future period is 
Significantly lower than the level of production in the 
most recent PeLLod, the learning curve model 
underestimates unit costs by approximately 17.0%. When 
the future production level iS comparable with the 
production level in the most recent period, the learning 
curve model underestimates unit costs by approximately 
4.9%. Finally, when the future production level is 
Significantly higher than the level of production in the 
most recent period, the learning curve model will 
overestimate unit costs by approximately 23.6%. 


10. Fixed-variable model cost prediction bias is influenced 
by the level of production in a future period. There is 
a negative relationship between the level of production 
in a future period (relative to the most recent period) 
and fixed-variable model bias. When the future 
production level is significantly lower, the fixed- 
variable model overestimates unit costs by approximately 
9.1%. When the future production level is relatively 
stable, the fixed-variable model overestimates unit costs 
by approximately 26.8%. Finally, when the future 
production level is significantly higher, the fixed- 
variable model overestimates unit costs by approximately 
6.0%. 


11. Bemis production rate adjustment model cost prediction 
bias is not significantly related to any of the variables 
included in the study. 

The selection of a particular cost progress model for 
estimating airframe unit costs depends primarily on the 
availability of required data and the cost versus the benefits 
of collecting the additional data required to employ more 
sophisticated cost prediction models. The findings from this 
study indicate that the learning curve, Bemis and random walk 
models all produce cost predictions which have low biases. 
However, the models differ widely in terms of their 


susceptibility to vagaries in the production process. Ceteris 


oe 


paribus, the Bemis model is superior to the other cost 
progress models because it not only exhibits the smallest bias 
but also is not significantly influenced (in terms of bias) by 
variations in the factors considered in this study. Hence, it 
provides the most robust and consistent cost estimates. 
Conversely, the bias of unit cost predictions obtained with 
the random walk, learning curve, and fixed-variable models is 
Significantly influenced by variations im the produ@wae. 
process. Consequently, the findings presented above should be 


considered when employing these less sophisticated models. 


B. FUTURE RESEARCH OPPORTUNITIES 
The results obtained in the study suggest other potential 


directions for future reseanen- 


1. The current study could be extended to include aircraft 
engines and missile propulsion systems to determine 
whether the findings concerning model prediction bias 
hold when the models are used to predict unit costs for 
these systems. 


2. Further studies could be conducted to determine the exact 
nature of the relationships between alrcraLe 
mission/missile mission, modification status, type of 
system (aircraft or missile) and learning curve model 
prediction bias. 
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